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9. A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local 
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Notice of Allowability 



Application No. 

10/629,403 

Examiner 

JASMINE STOKELY-COLLINS 



Applicant(s) 

LIU ET AL. 
Art Unit 

2423 



~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 

All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication will be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1 .313 and MPEP 1 308. 

1 . ^ This communication is responsive to RCE and IDS filed 1/7/20 1 1 . 

2. |EI The allowed claim(s) is/are 1-13,16-17,20-21,24-32,34-35,37-46 . 

3. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a) □ All b) □ Some* c) □ None of the: 

1 . □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

4. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration is deficient. 

5. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draftsperson's Patent Drawing Review ( PTO-948) attached 

1) □ hereto or 2) □ to Paper No./Mail Date . 

(b) Q including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1 .84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

6. □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 

attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL. 
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EXAMINER'S AMENDMENT 

1 . An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1 .31 2. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview 
with Nathan Feld on 3/8/201 1 . 

The application has been amended as follows: 

Cancel claims 18-19 and 22-23. 
Claim 20: 

Line 1 . Change "The method of claim 18 wherein:" to - The method of claim 12 
wherein: - . 

Claim 21: 

Line 1 . Change "The method of claim 1 8 wherein:" to - The method of claim 1 0 
wherein: --. 



Claim 24: 

A shared interactive environment, comprising: 
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a camera system to provide a first live view of a location and a second live view 
of the location, wherein the second live view can be configured to zoom in on a portion 
of the first live view, and wherein each live view shows a different view of a plurality of 
physical devices at the location; 

a first graphical user interface (GUI) coupled to the camera system which 
presents the first live view and the second live view, wherein each view shows one or 
more of the plurality of physical devices at the location , wherein information is 
associated with at least one physical device in the first live view : 

a device controller to dynamically control the physical device in response to 
interaction of a first user with the GUI; 

wherein the interaction can include annotating at least one of: 1) the first live view 
of the location; and 2) the second live view of the location; 

wherein annotations are automatically transferred to the physical device in the 
live views if the annotation is at least partially drawn over the physical device as it 
appears in the live video image, and wherein the annotation is displayed on the physical 
device such that the annotation can be viewed at the location; 

wherein interaction can include causing the information to be transferred from the 
at least one physical device to a second physical device in the second live view, 
wherein the transfer is brought about by manipulating a visual representation of the 
information by interacting with the first live view and the second live view; 

a device tracker coupled to the camera system and to dynamically recognize new 
physical devices; and 
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wherein the camera system can be mounted on a mobile, robotic platform. 

Allowable Subject Matter 

2. Claims 1 -1 3, 1 5-1 7, 20-21 , 24-32, 34-35, and 37-46 are allowed. 

3. The following is an examiner's statement of reasons for allowance: 
Regarding independent claim 1 , the prior art fails to teach or reasonably suggest 

a method for exchanging information in a shared interactive environment, comprising: 
selecting a first remote physical device in a first live video image, that shows a 
first view of the shared interactive environment, wherein information is 
associated with the first remote physical device; 

causing the information to be transferred to a second remote physical device 
shown in a second live video image that shows a second view of the shared 
interactive environment, wherein the transfer is brought about by manipulating a 
visual representation of the information shown in the first live image by 
interacting with the first live video image and the second live video image. 

Regarding independent claim 10, the prior art fails to teach or reasonably 
suggest a method for exchanging information in a shared interactive environment, 
comprising: 

selecting a first object wherein the first object is one of 1) a first physical device 
at a remote location shown in a live video image, and 2) an icon on a computing 
device, wherein the live video image shows a view of the remote location which 
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includes a plurality of physical devices including the first physical device; 
causing information associated with the first object to be transferred to a second 
object wherein the second object is the other of 1) the first physical device at the 
remote location shown in the first live video image, and 2) the icon on the 
computing device; 

wherein the transfer is brought about by manipulating a visual representation of 
the information in the live video image; 

wherein manipulating includes interacting with the first object in the live video 
image and the second object. 

Regarding independent claim 24, the prior art fails to teach or reasonably 
suggest a shared interactive environment, comprising: 

a camera system to provide a first live view of a location and a second live view 
of the location, wherein the second live view can be configured to zoom in on a portion 
of the first live view, and wherein each live view shows a different view of a plurality of 
physical devices at the location; 

a first graphical user interface (GUI) coupled to the camera system which presents the 
first live view and the second live view, each view shows one or more of the plurality of 
devices at the location, wherein information is associated with at least one physical 
device in the first live view; 

a device controller to dynamically control the physical device in response to interaction 
of a first user with the GUI wherein the interaction can include annotating at least one 
of: 1 ) the first live view of the location; and 2) the second live view of the location; 
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wherein interaction can include causing the information to be transferred from 
the at least one physical device to a second physical device in the second live 
view, wherein the transfer is brought about by manipulating a visual 
representation of the information by interacting with the first live view and the 
second live view. 

Regarding independent claim 37, the prior art fails to teach or reasonably 
suggest a non-transitory computer readable memory having instructions stored thereon 
that when executed by a processor cause a system to: 

select a first remote physical device in a first live video image, that shows a first 
view of the shared interactive environment, wherein information is associated 
with the first remote physical device; and 

cause the information to be transferred to a second remote physical device 
shown in a second live video image that shows a second view of the shared 
interactive environment, wherein the transfer is brought about by manipulating a 
visual representation of the information by interacting with the first live video 
image and the second live video image. 

Regarding independent claim 46, the prior art fails to teach or reasonably 
suggest a system comprising: 

means for selecting a first remote physical device in a first live video image that 
shows a first view of the shared interactive environment, wherein information is 
associated with the first remote physical device; and 

means for causing the information to be transferred to a second remote physical 
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device shown in a second live video image that shows a second view of the 
shared interactive environment, wherein the transfer is brought about by 
manipulating a visual representation of the information by interacting with the 
first live video image and the second live video image. 

Any comments considered necessary by applicant must be submitted no later 
than the payment of the issue fee and, to avoid processing delays, should preferably 
accompany the issue fee. Such submissions should be clearly labeled "Comments on 
Statement of Reasons for Allowance." 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JASMINE STOKELY-COLLINS whose telephone 
number is (571 ) 270-3459. The examiner can normally be reached on M-F 9:30-5:30 
EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andrew Koenig can be reached on (571) 272-7296. The fax phone number 
for the organization where this application or proceeding is assigned is 571 -273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-91 99 (IN USA OR CANADA) or 571 -272-1 000. 



/Jasmine Stokely-Collins/ 
Examiner, Art Unit 2423 

/Andrew Y Koenig/ 

Supervisory Patent Examiner, Art Unit 2423 



